From what I tested - models on iOS work. Had some issues with building LLM demo app due to audio-api throwing an error - maybe a version bump would help?
Will test Android tomorrow

I tested Android - no issues found. I think we can merge it safely.

@barhanc one small thing. Could you test if Bielik model from our HuggingFace works on android? If you get: E Missing operator: [12] torchao::_linear_8bit_act_4bit_weight.out would you be able to rebuild android libs so they have custom torchao ops?

Or maybe the reason is that the model was exported with ExecuTorch 1.0.0 and now is not compatible. Since most (if not all) of the re-exported llms uses 8 bit activation 4 bit weights it's strange that this particular model does not work on android.

EDIT: I successfully built this op in arm64 and model ran on android (device that supports ARMv8.4+, e.g. One+ 12) and later for ARMv8.2+ np. FYI: the changes for version ARMv8.4+ are on the first commit and for ARMv8.2+ on the second commit on this branch: https://github.com/software-mansion/react-native-executorch/tree/msluszniak/torchao-4bit-android

@barhanc one small thing. Could you test if Bielik model from our HuggingFace works on android? If you get: E Missing operator: [12] torchao::_linear_8bit_act_4bit_weight.out would you be able to rebuild android libs so they have custom torchao ops?

Or maybe the reason is that the model was exported with ExecuTorch 1.0.0 and now is not compatible. Since most (if not all) of the re-exported llms uses 8 bit activation 4 bit weights it's strange that this particular model does not work on android.

EDIT: I successfully built this op in arm64 and model ran on android (device that supports ARMv8.4+, e.g. One+ 12) and later for ARMv8.2+ np. FYI: the changes for version ARMv8.4+ are on the first commit and for ARMv8.2+ on the second commit on this branch: https://github.com/software-mansion/react-native-executorch/tree/msluszniak/torchao-4bit-android

The problem with Bielik was connected to the previous export of the model. I've re-exported the model and updated it on HF. It should work now without requiring the fix.